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Abstract: During past few years, people have been substantially attracted towards Content-Based Image 
Retrieval (CBIR) because of its varied multimedia applications. CBIR is one of the most popular research areas 
of digital image processing. The goal of the CBIR is to extract the visual features of the image such as color, 
texture or shape. An attempt is made to develop a Sketch Based Image Retrieval (SBIR) making it use of the 
features such as shape or form of the object [1]. This paper aims to introduce the creation and design of SBIR 
system making use of extraction techniques of Biased Maximum Margin Analysis (BMMA) and a Semi- 
Supervised Biased Maximum Margin Analysis (Semi BMMA) [2].. With the help of existing methods, design a 
task specific descriptor, which can handle the informational gap between a sketch and a colored image. The 
result of SBIR includes the set of positive images (relevant) and negative images (irrelevant). Iterative process 
acts on the positive set for the optimal extraction of the object. By using Laplacian regularizer to BMMA, the 
Semi BMMA integrates the information of unlabelled samples [2] to result in the better extraction of refined set 
of objects. The SBIR have several applications such as digital libraries, crime prevention, photo sharing sites 
etc. An important application is a matching a forensic image to gallery of mug shot images. 
Index Terms: CBIR, SBIR, BMMA, Semi BMMA. 



I. Introduction 

In the recent years, CBIR has attracted the attention because of its varied multimedia application. It is 
initiated by the immense growth of image records and online accessibility of remotely stored images. The 
traditional search scheme requires the huge image database. In the textual based image retrieval, we can search 
flexibly using key words but if we use image, we can't apply dynamic methods. Two questions can arises. The 
first is who yields the keywords. And second is an image can be efficiently represented by keywords. 

In most of the cases, if we want to search correctly, some data have to be recalled. In this case, we 
search the images using some features and these features are called keywords. At this time unfortunately they 
are not frequently used retrieval systems which retrieve the images using visual information of a sample image. 
One reason for this is, it is too difficult to give some unique and identifiable information to a text. 

In CBIR, an image is represented using some example images and some low-level visual features. They 
are color, texture , shape, etc. These visual features are extracted to represent the images in the database. This 
low-level features extracted from images may not accurately detecting the high-level features. 

To reduce the gap between this low-level visual features and high-level semantic concepts, Relevance 
Feedback (RF) scheme is introduced to improve the performance of CBIR. One of the RF technique is Support 
Vector Machine (SVM). Different versions of SVMs are developed. One class SVM computes the density of 
positive feedback samples. After this, a biased SVM is developed. It overcomes the difficulties of one class 
SVM. Varied learning methods are implemented to find intrinsic structure of images and it enhances the 
performance. With the observation that "all positive are alike; each negative example is negative in its own 
way", a biased learning problem is developed. In this, there is some unknown number of classes, user is only 
concentrated with positive class. However, all these method has its own drawbacks. 

Two class SVM is one of the method used in the past few years. In the RF scheme, user labels number 
of relevant samples as positive feedbacks and number of irrelevant samples as negative feedbacks. The 
traditional SVM treats positive and negative feedbacks equally .The problem here is that, different semantic 
concepts belong to different subspaces and each image can lie in many different subspaces. It reduces the 
performance of the CBIR. In addition, the traditional SVM is not considering the information of the unlabelled 
samples too. 

This paper aims to develop SBIR system which can retrieve images using sketches. The user has a 
drawing area where the user can draw the sketches, which are the base of retrieval method. 

The retrieval system in the SBIR provides response list to the displaying system. It contains positive 
and negative feedbacks. That means, from the retrieved images, user labels the number relevant samples as 
positive feedbacks and number of irrelevant samples as negative feedbacks. For extracting positive feedbacks, 
the techniques called Biased Maximum Margin Analysis (BMMA) and a Semi- 
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Supervised Biased Maximum Margin Analysis (Semi BMMA) are used. These schemes are mainly based on the 
concept that it treats the positive and negative feedback unequally. Also, labeled positive feedbacks are mapped 
as close as possible and negative feedbacks are separated from positive feedbacks by a maximum margin in 
reduced subspace. 

II. Related Works 

Content based image retrieval is a process used for searching the relevant images based on user input. 
The input could be parameters, sketches or example images. A traditional CBIR system firstly extracts the 
image features and stored them. Then it compares with image features of database images and displays the 
results. Feature extraction and similarity measure are very dependent on the features used. Histogram is the most 
commonly used technique to describe image. 



Imae^ Query 



Result 



1 v ; VI 1 L 1 Ii — 



Feature 
Extraction 



Featnrre 
Vector 



SiivulfLrity 
C* omput.i ti 011 



Fig 1: Architecture of CBIR process. 

Fig 1 describes the flow of typical CBIR process although content based methods are efficient, they cannot 
always match user's expectations. 

Relevance Feedback (RF) techniques are used to adjust the query by user's feedback. RF is a powerful 
technique used in traditional CBIR system. It is a process of automatically adjusting an existing query using 
information fed back by the user about the relevance of previously retrieved objects such that the adjusted 
query. The issue in relevance feedback is how to utilize the feedback information for improving the retrieval 
performance. After obtaining retrieval results, user provide the feedback for checking results are relevant or not 
relevant. If the results are non relevant, the feedback loop is repeated many times until the user is satisfied. 
The basic idea behind the RF scheme is to shift the burden of finding the right query formulation from the user 
to the system. In order to make this true, the user has to provide the system with some information, so that 
system can perform well in answering the original query. 

The concept of RF was introduced into CBIR from the concept of text based information retrieval in 
the 1998's and then has become a popular technique in CBIR 

When a user submits the query image. It firstly extracts the feature vectors and comparing with the 
feature vectors of database images. Then the retrieval results displays to the user. Then user gives the feed back 
in the form of relevance judgements expressed over retrieval results. The relevance judgements evaluate the 
results based on a three value assessments [3]. These three values are relevant, non relevant and don't care. 
Relevant means, the image relevant to the user, non relevant means, the image definitely not relevant and don't 
care means the user doesn't say anything about that image. If the user feedback is relevant, then feedback stops. 
Otherwise it continues until users get satisfied with that result. 

The first RF based approach to CBIR in which user and system interact to refine high level queries into 
low level visual features which addresses the gap between high level semantic concepts and low level visual 
features. The evaluation parameter used was convergence ratio[4]. 

Learning in CBIR system is divided into short term learning and long term learning. Most of the 
researches are focused on query tuning in a single retrieval session. This is commonly known as intra-query 
learning or short term learning. In contrast, inter-query, also known as long term learning is strategy that 
attempts to analyze the relationship between the current and past retrieval sessions. 
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In short term learning, only the feedbacks of the current search session are used in the learning algorithm. The 
main challenge in this approach is to find the best combination of image features that presents the users query. 
Such optimum set of feature include features that capture similarities between positive images. So short term 
learning include support vector machine, Bayesian learning, discriminant learning and so on. 
Long term learning utilizes the feedbacks collected during prior retrieval session. It is a cumulative process for 
collecting fast feedbacks and which are stored in the form of matrix. 

Bayesian network as a relevant image that include short term learning, is an adoption model to select a 
number of good points composing the positive feedback information. It is a powerful tool that is applicable in 
the point of view of relevance feedback in image retrieval. The precision rate as a function of the number of 
feedback iterations was used as the performance measure. 

A relevance feedback approach based on Bayesian classifier. For positive examples, a Bayesian 
classifier was used to determine the distribution of the query space. The image set used was the Corel Image 
Gallery [7]. The images of the same category as the query example were considered as relevant to the query. 
Positive feedback process was running as following: images from the same category as the query example that 
were ranked in top 50 were assigned as the positive feedbacks. For negative feedback process the first two 
irrelevant images were assigned as negative examples. 

Support Vector Machine (SVM) is the one of the RF based approach. [8] The traditional SVM is binary 
classifier. A svm captures the query concept by separating the relevant images from irrelevant images with the 
hyper plane in a projected space, usually high dimensional one. The projected points on the one side of the 
hyper plane are considered relevant to the query concept and rest are irrelevant. 




Fig 2: SVM Hyper planes 

This SVM have mainly two drawbacks. First it treats positive and negative samples equally which is 
not appropriate, since the 2 groups of training feedbacks have distinct properties. Second most of the svm based 
RF techniques don't take the unlabelled samples. To overcome these draw backs, BMMA and SemiBMMA are 
introduced in this paper. 

III. Problem Domain 

CBIR is a system for retrieving an image from the database using visual information such as color, 
texture, or shape. Before the spreading of information technology a huge number of data had to be managed, 
processed and stored. It was also textual and visual information. Parallelly of the appearance and quick 
evolution of computers an increasing measure of data had to be managed. The growing of data storages and 
revolution of internet had changed the world. The efficiency of searching in information set is a very important 
point of view. In case of texts we can search flexibly using keywords, but if we use images, we cannot apply 
dynamic methods. Two questions can come up. The first is who yields the keywords and the second is an image 
can be well represented by keywords. In many cases if we want to search efficiently some data have to be 
recalled. The human is able to recall visual information more easily using for example the shape of an object, or 
arrangement of colors and objects. Since the human is visual type, we look for images using other images, and 
follow this approach also at the categorizing. 
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In this case we search using some features of images, and these features are the keywords. At this moment 
unfortunately there are not frequently used retrieval systems, which retrieve images using the non-textual 
information of a sample image. What can be the reason? One reason may be that the text is a human abstraction 
of the image. To give some unique and identifiable information to a text is not too difficult. At the images the 
huge number of data and the management of those cause the problem. The processing space is enormous. 

This paper is about to develop sketch based image retrieval system, which can retrieve using a sketches 
in frequently used database. The user has a drawing area where he can draw sketches, which are the base of the 
retrieval method. 

IV. Problem Identification 

The Sketch Based Image Retrieval (SBIR) was introduced in QBIC and Visual SEEK systems. In these 
systems, user draws the colour sketches and blobs on the drawing area. The images were divided into grids and 
the colour and texture features were determined in these grids. The application of the grids was also used in 
other algorithms, for example in the edge histogram descriptor (EHD) method. The disadvantage of this method 
is that they are not invariant opposite rotation, scaling and translation. Lately the development of difficult and 
robust descriptors is emphasized. Another research approach is the application of fuzzy logic or neural 
networks. In these cases the purpose of the investment is the determination of suitable weights of image 
features. 

Even though the measure of research in the sketch based image retrieval increases, there is no widely 
used system sketch based image retrieval system. Our goal is to develop content based image retrieval system 
which databases are available for anyone looking back to free hand drawing. The user has a drawing area where 
he can draw all the shapes and moments which are expected to occur in the given location and with a given size. 
The retrieval results are grouped by color for better clarity. Our most important task is to bridge the information 
gap between the drawing and the picture, which is helped by own preprocessing and transformation process. In 
our system the iteration of the utilization process is possible, by the current results looking again, thus increasing 
the precision. 

V. Problem Statement 

This paper aims to develop SBIR system which can retrieve similar relevant images using BMMA and 
SemiBMMA techniques. 



VI. Proposed System 

In the SBIR system, firstly the user draws a sketch or loads an image. When the drawing has been finished or 
appropriate representative has been loaded, the retrieval process is started. The retrieved image is first 
preprocessed. After that the feature vector is generated, then using the retrieval system a search is executed in 
the previously indexed database. The retrieving process is done using BMMA and SemiBMMA process. As a 
result of searching, result set is displayed to the user, which appears in the user interface on a systematic form. 

6.1 Global Structure of the SBIR 
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Fig 3: Global structure of the system 



www.ijeijournal.com 



Page I 41 



Sketch Based Image Retrieval Using BMMA and SEMI-BMMA 



The global structure of the system contains preprocessing system, displaying system, feature vector extraction 
system, retrieval system and database system. 

A. Preprocessing System 

Preprocessing system which eliminates the problem caused by the diversity of images. The input of the 
preprocessing system is one sketch image and output is the processed result set. The system was designed for 
the database containing relatively simple images but even in such cases large difference can occur among 
images in file size or resolution. 

B. Feature Vector Extraction System 

Feature vector stores the feature vectors of the database images. Mainly edge histogram method is used 
in this system. One problem was encountered in the extraction of features in the hand drawing sketches. An 
information gap arise between retrieved sketch and coloured image of the database. This is avoided by the 
images in the database is converted into images. 

C. Database Management System 

Data base system contains three modules. The storage, data manipulation and retrieval. The storage 
module provides images, information and the associated feature vectors are uploaded to database. The file name, 
size and format of the image are attached. The retrieval results are obtained by usage of query module. The 
retrieval results contacts the database which provides the descriptors. 

D. Retrieval System 

As the feature vectors are ready, the retrieval can start. The output of the retrieval system is the 
response list of the query image. 

E. Displaying System 

The displaying system provides response list to the user. It contains positive and negative feedbacks. 
That means, from the retrieved images, user labels the number relevant samples as positive feedbacks and 
number of irrelevant samples as negative feedbacks. For extracting positive feedbacks, the techniques called 
Biased Maximum Margin Analysis (BMMA) and a Semi-Supervised Biased Maximum Margin Analysis (Semi 
BMMA) are used. These schemes are mainly based on the concept that it treats the positive and negative 
feedback unequally. Also, labeled positive feedbacks are mapped as close as possible and negative feedbacks 
are separated from positive feedbacks by a maximum margin in reduced subspace. 



6.2 Proposed Scheme Structure 
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Fig 4: Sketch of Proposed scheme. 
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In this Fig 4 when a image query is submitted by the user, i.e, user draws an image or loads the image 
to the system. It firstly extracts its low-level visual features. Then all the images are arranged on the basis of 
similarity metric i.e. Euclidian distance. If the user is satisfied with the resulting image, then the process is 
completed and the results are displayed to the user. 

Most of the time, the user is not satisfied with the first results because of the semantic gap. Then the 
user will label the most relevant images as positive feedbacks in the results. All the remaining images are 
labeled as negative feedbacks by the system. Based on positive and negative feedbacks, RF model can be trained 
on the basis of various existing algorithms. Then all the images are re-arranged on the basis of new similarity 
metric. After each round of iteration, the user will check whether the results are satisfied or not. If so, the 
process is completed otherwise it repeats until the user is satisfied with the expected results. 

6.3. BMMA and Semi BMMA 

The positive and negative feedbacks have distinct properties for CBIR. The SVM based RF treats 
positive and negative feedbacks equally. One problem is that different semantic concept lie in different semantic 
subspaces and each image can lie in any different concept subspaces. To avoid this problem, we use BMMA for 
SVM RF scheme. In this, negative feedbacks which have different concept with query image are separated by 
maximum margin from positive feedbacks which share similar concept with query image. So we can easily map 
the positive and negative feedbacks onto semantic subspace in accordance with human perception of image 
contents. 

Using Laplacian regularizer to the BMMA, we can easily use the information of unlabelled samples in 
database, which is called SemiBMMA for the SVM RF. The result of the Laplacian regularizer is based on the 
local consistency which was inspired by the recently emerging varied learning community and can effectively 
depict the weak similarity between unlabelled samples. 

Then the remaining images in the database are projected on the semantic subspace and a similarity 
measure is applied to sort the images. For SVM based RF, the distance to the hyper plane is the criterion to 
separate query relevant samples i.e. positive feedbacks from the query irrelevant samples i.e. negative 
feedbacks. After the projection step, all the positive feedbacks are clustered together while negative feedbacks 
are well separated from positive feedbacks by a maximum margin. So the resultant SVM classifier is much 
simpler and better than in the original high dimensional feature space. 

BMMA aims to learn a projection matrix 'a ' such that in the projected space, the positive samples have 
high local within-class similarity, but the samples with different labels have high local between-class 
separability. 

6.4 Mathematical Concept of BMMA and SemiBMMA 

In each round of feedback iteration, there are n samples P = {pipj pje R h - For simplicity, we 
assume that the first n + samples are positive feedbacks, p i (l<i<n + ), the next ri samples are negative feedbacks 
Pi(n + +1 <i <n + + ri), and all others are unlabelled samples p;(n + +ri +1 <i< n). 

Let l(pi) be the class label of sample P,. We denote l(pj=l for positive feedbacks, l(pj)= -1 for negative 
feedbacks and l(p,)=0 for unlabelled samples. To better show the relationship between the proposed approaches, 
we use the similar notations and equations in graph embedding frame work. 

Firstly, two different graphs are formed: 1) the intrinsic graph G, which characterizes the local similarity of 
feedback samples, 2) the penalty graph CP, which characterizes the local discriminant structure of the feedback 
samples. 

For all positive feedbacks, we first compute the pair wise distance between each pair of positive 
feedbacks. Then for each positive feedback P„ we find its k t nearest neighbourhood positive feedbacks, which 
can be represented as a sample set Nf, and put an edge between Xj and its neighbourhood positive feedbacks. 
Basically, the intrinsic graph measures the total average distance of the \N"\ nearest neighbourhood sample pairs, 
and is used to characterize the local within-class compactness for all the positive feedbacks. 

For the penalty graph CP, its similarity matrix represents geometric or statistical properties to be 
avoided and is used as a constraint matrix in the graph embedding framework. In the BMMA, the penalty graph 
is constructed to represent the local separability between the positive class and the negative class. More strictly 
speaking, we expect that the total average margin between the sample pairs with different labels should be as 
large as possible. 

For each feedback sample, we find its k 2 neighbour feedbacks with different labels and put edges 
between corresponding pairs of feedback samples 

For each unlabelled samples pj(n + + ri +1 <i < n), for each unlabelled samples we find its ki nearest unlabelled 
samples which can be represented as a sample set N" , and put an edge between the unlabelled /?, and its 
neighbourhood unlabelled samples. This matrix term is known as Laplacian matrix. 

The motivation for introducing this term is inspired by the regularization principle, which is the key to 
enhancing the generalization and robust performance of the approach in practical applications. There are lot of 
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possible ways to choose a regularizer for the proposed BMMA. In this work we chose the Laplacian regularizer, 
which is largely inspired by the recently emerging manifold learning community. Actually, this scheme can 
preserve weak (probably correct) similarities between all unlabeled sample pairs and thus effectively integrate 
the similarity information of unlabeled samples into the BMMA. By integrating the Laplacian regularizer into 
the supervised BMMA, we can easily obtain the Semi BMMA for the S VM RF. The difference between BMMA 
SVM and SemiBMMA SVM for two classes of feedbacks can be shown in Fig:5 



SemiBMMA SVM BMMA SVM 




positive samples 
negative samples 
unlabelled samples 
uakbelkd samples; 



Fig 5: SVM hyper plane comparison between BMMA SVM and SemiBMMA SVM for two classes of feed 

backs 



6.5 SemiBMMA Algorithm 

Input: P = {piP2 pje R h > stand for all feedback samples and unlabelled samples which include the 

positive set P + , the negative sample set P" , and unlabelled samples 

Step 1 : Construct the supervised intrinsic graph G and calculate the matrix value of positive feedbacks 

Step 2: Construct the supervised penalty graph G p and calculate the matrix value of negative feed backs. 

Step 3: Construct the Laplacian regularizer and calculate the matrix value of unlabelled samples 

Step 4: Calculate the projection matrix a : project all positive , negative and remaining samples in the database 
on to the reduced subspace respectively. 

Output: Positive and Negative samples Y + and Y" , in this reduced subspace. 

VII. Simulation 

The proposed work has been implemented in MATLAB. MATLAB (matrix labortary) is a numerical 
computing environment. MATLAB (matrix laboratory) is a numerical computing environment and fourth 
generation programming language. Developed by math works, MATLAB allows matrix manipulations, plotting 
of functions and data, implementation of algorithms, creation of user interface, and interfacing with programs 
written in other languages, including C, C++, Java, and Fortran. we can also use MATLAB to analyze and 
visualize data using automation capabilities, thereby avoiding the manual repetition common with other 
products. Programming and developing algorithms is faster with MATLAB than with traditional languages 
because MATLAB supports interactive development without the need to perform low-level administrative tasks, 
such as declaring variables and allocating memory. Thousands of engineering and mathematical functions are 
available, eliminating the need to code and test them yourself. At the same time, MATLAB provides all the 
features of a traditional programming language, including arithmetic operators, flow control, data structures, 
data types, object-oriented programming, and debugging features. MATLAB helps you better understand and 
apply concepts in a wide range of engineering, science, and mathematics applications, including signal and 
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image processing, communications, control design, test and measurement, financial modeling and analysis, and 
computational biology. 

7.1 MatLab Key Features in the model 

• High-level language for numerical computation, visualization, and application development 

• Interactive environment for iterative exploration, design, and problem solving 

• Mathematical functions for linear algebra, statistics, Fourier analysis, filtering, optimization, numerical 
integration, and solving ordinary differential equations 

• Built-in graphics for visualizing data and tools for creating custom plots 

• Development tools for improving code quality and maintainability and maximizing performance 



VIII. Input Output Model 
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IX. Data Flow Model 
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Fig 6: Data Flow Model 

In Fig 6, when user draws an image or loads an image, the system firstly extracts the feature vector of sketch 
image. These feature vectors is compared with the feature vector of database images with the help of BMMA 
and SemiBMMA. As a result, we can get the group of images which is similar to the sketch image or query 
image. If the user is satisfied with this results, then the process is completed otherwise user gives the feedback to 
the system. Again the process is continued based on the positive feedbacks. After this, final result is presented to 
the user that contains user relevant images. 

X. Result Analysis 

10.1 Referenced database 

The Microsoft Research Cambridge Object Recognition Image database is used in this project. This database 
contains 209 realistic objects. All objects have been taken from 14 different orientations with 450 450 
resolution. This database is widely used in computer and psychology studies. Some images of the database can 
be shown in the fig 7. 
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Fig 7: Database 

20.2 Conceptual Set Model 

To understand the SBIR concept clearly, consider the integer number system (1-100) as database. The 
user draws number '2' as sketch image and loads to the SBIR system. Resultant set of system consists of 
2, 12,20. ...29,32,42. ...92. In this set, user label 2,20,21,22 as positive images and loads to the system. Then 
2,20,21,22. ..29 are the first 12 image of the final result. 



Database: 



1,2, 3. ...100 




Sketch Image 



Feedback: 




First iteration result set: 



2, 12, 20, 21.... 29, 32 
42... 92 




Final Result set: 



2,21,22,23,24, 25,26 
29, .... 




XI. Conclusion 

CBIR is one of the research areas in digital image processing. Among the objectives of this paper 
performed to design, implement and test a sketch-based image retrieval system. Two main aspects were taken into 
account. The retrieval process has to be unconventional and highly interactive. The robustness of the method is 
essential in some degree of noise, which might also be in case of simple images. SVM based RF technique is one of 
the popular method to bridge the semantic gap and it increases the performance of the CBIR. In this we use BMMA 
and SemiBMMA technique for extracting the positive feedbacks of the user which enhances the performance of the 
sketch based image retrieval. 
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